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BE > 
CASE STUDY 
TRUST THE ALGORITHM 
OR YOUR GUT? 


A VP DECIDES WHICH CANDIDATE 
TO PROMOTE. BY JEFFREY T. POLZER 


Aliyah Jones was having 
trouble paying attention 
to the farewell toasts. 
Although she was sad 


EA 
CASE STUDY 
CLASSROOM NOTES 


Companies use algorithms 
in people-related decisions 
for many reasons, 
including consistency, 
reduced bias, casting 

a broader net, and 
efficiency. How might 

the recommendations an 
algorithm makes differ 
from those of a hiring 
manager who is not 

using data analytics? 


to see her longtime colleague Anne Bank go, she was more consumed 
with tryingto figure out who should replace her. 

As a VP of sales and marketing for Becker-Birnbaum International, a 
global consumer products company, Aliyah knew she needed a talented 
marketing director to support her division’s portfolio of 34 products. 
After working with HR to narrow down the list of candidates, she had two 
finalists, both internal: Molly Ashworth, a brand manager on her team in 
the cleaning division, and Ed Yu, a rising star from BBI’s beauty division. 

Aliyah liked Molly and respected her work. Two years earlier, Molly 
had spearheaded a new subscription service for BBI cleaning products, 
which had shown strong growth in the past two quarters. Customers 
seemed to love the convenience, and the R&D, marketing, and executive 
teams had gotten excited about the service as a platform to test new 
offerings. Having mentored Molly through the pitch and launch of the 
service, Aliyah was intimately familiar with her protégé’s strengths and 
weaknesses and was certain that she was ready for the next challenge. 

But soon after the position had been posted, Christine Jenkins, a 
corporate VP of HR, had come to Aliyah with Ed’s résumé. Like Molly, 

Ed had joined BBI right out of business school and been quickly tapped 

as a high potential. He also had his own BBI success story: As a brand 
manager in the beauty group, he had revived its 20-year-old FreshFace 
makeup-removal product line, increasing sales by 60% in three years. 
Perhaps more important to Christine, he’d been recommended as a 
96% match for the job by HR’s new people-analytics system, which 
she had championed. (Molly had been an 83% match.) The goal 

of the initiative was to expand the use of data analytics to human 
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CASE STUDY TRUST THE A 


Does using algorithms for 
any type of people analytics 
violate employees” privacy? 
New laws—in particular, 
the EU's General Data 
Protection Regulation 
(GDPR)—are setting 

limits on what information 
employers can and cannot 
collect, and how employees 
must be notified. 


Research shows that 

hiring managers typically 
form opinions about a 
candidate's personality and 
competence in the first 30 
seconds of an interview. 


e 


LGORITHM OR YOUR GUT? 


O“ 


resources, to inform 
hiring, promotion, 
and compensation 
decisions. Aliyah was glad to see two 
insiders in contention—she’d come up 
the ranks herself—but that made the 
decision harder. 

As the COO made a toast to Anne, 
Aliyah considered her interviews with 
Ed and Molly. 


MEETING ED YU 


“Tm sorry Pm so late,” Ed said, looking 
a little discombobulated. “My Uber 
driver insisted he knew a shortcut from 
Heathrow—but he was wrong.” 

Aliyah couldn't help drawing an 
immediate comparison with Molly, 
who was always steady and calm, but 
she tried to keep an open mind. 

“No problem,” she said. “Shall we 
get started?” 

“Absolutely,” Ed said eagerly. 

“What interests you aboutthejob?” 

Ed explained that while he was 
proud ofthe growth FreshFace had seen 
under his leadership, he was ready for 
a new challenge. He’d enjoyed diving 
deep into one product but felt his 
skills were better suited for a position 
that would allow him to work across 
programs and direct a larger portfolio. 

Sharp, clear answer, Aliyah thought. 
“What have you learned in beauty that 
would apply in cleaning?” she asked. 

This was an important question. 
BBI’s top team had directed the 
divisions to share more best practices 
and improve collaboration. In fact, her 
boss wanted herto work more closely 
with her peers in other divisions. 

Ed explained how he thought 
his division’s approach to in-field 
customer research, which he credited 
with boosting FreshFace sales, could 
work in cleaning. Partnering with 
anthropologists was something 
Aliyah’s team had talked about but 
hadn’t yet tried out. 

He also asked about the 
new subscription program, 

referencing a recent 

white paper on trends 

in subscription business 
models. He'd clearly done 
his homework, was smart and 
ambitious, knew BBI’s business 
well, and seemed eager to 
learn. But his answers and 


even his questions seemed a bit stiff. 
Aliyah didn’t sense the dynamism or 
entrepreneurial mindset that she knew 
Molly had. Maybe he’s nervous, she 
thought. Or maybe that’s just who he is. 
Aliyah didn’t doubt Ed could do the 
job. But she didn’t feel excited 
about hiring him. 


MOLLY’S “INTERVIEW” 


Setting Molly’s interview 
up for the same day as Ed’s 
had seemed like a great idea 
when she’d suggested it to 
Christine, and given the noon 
time slot, it had been only natural to 
meet at their usual lunch spot near the 
office. But as soon as Aliyah walked into 
the café, she realized how unfair these 
back-to-backs were to Ed. 

It was impossible not to hug Molly 
hello and ask for a quick update on her 
projects and family. They even ordered 
the same thing: curried egg salad. But 
as soon as the waitress left, Molly got 
down to business: “I know we e-mail 
10 times a day, but Pd like to treat this 
as a formal interview.” 

Aliyah smiled. “Of course.” 

As Christine had advised her to do, 
she asked questions that were the same 
or similar to the ones she’d asked Ed. 

“Tell me why you're interested in 
this job,” she started. It was awkward. 
Aliyah knew the answer already, but 
to Molly's credit, she proceeded asif 
they weren’t close colleagues. With 
each response, she demonstrated deep 
knowledge of the business, and she 
had good ideas for collaborating across 
programs and building on the success 
of the subscription program. She was 
as polished and thoughtful as Ed, but 
she also seemed warmer and more 
self-aware. 

Knocked it out of the park, Aliyah 
thought, as they walked back to the 
office. Looking at the smile on Molly’s 
face, Aliyah knew her protégé was 
feeling confident that she’d done well. 


THE ALGORITHM 


The day after Anne’s farewell party, 
Aliyah met with Christine and Brad 
Bibson, a data scientist on the people 
analytics team. 

“T know you were leaning 
toward Molly after we debriefed the 
interviews,” Christine said, “but we 


Managers tend to 

hire people similar to 
themselves, studies show. 
For example, Kellogg 
School of Management 
professor Lauren Rivera 
found that managers prefer 
recruits who have the 
most potential to become 
friends, even over those 
who are more qualified. 
Should Aliyah worry that 
she’s choosing Molly 
because she likes her? 


Unstructured interviews 
are the default method 
for most hiring managers, 
but numerous studies 
have found them to be 
poor predictors of actual 
on-the-job performance. 
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Network analyses can 
reveal patterns that are 
otherwise hard to see—for 
example, by identifying 
which employees are 
most central to informal 
information flows. 


Studies have revealed 

a phenomenon called 
“algorithm aversion.” 
Even when data-driven 
predictions yield higher 
success rate than intuitive 
human forecasts, people 
often prefer to rely on the 
latter. And if they learn 

an algorithm is imperfect, 
they simply won't use it. 
Under what conditions 
would you base a decision 
on data analysis? 


Along with managers, 
many applicants are 
skeptical of algorithms, 
according to Pew. A 
majority of Americans 
(76%) say they would not 
want to apply for jobs that 
use a computer program 
to make hiring decisions. 
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wanted to share some 


more data” 

Brad handed over two colorful 
diagrams. “These are network analyses 
of Molly’s and Ed's e-mail and meeting 
history at BBI. With their permission 
and without looking at the content of 
their e-mails or calendars, we analyzed 
who they’d been in contact with across 
the firm over the past six months.” 

It was clear from the diagrams that 
Ed was connected to not just his beauty 
division colleagues but also key people 
in other groups. Molly’s network was 
mainly within cleaning products. 

“I didn’t know we were doing this 
kind of analysis,” Aliyah said. 

“We’ve just started looking at 
networks,’ Brad said, “and we think 
they can reveal useful insights.” 

“I know one chart isn’t going 
to sway your decision,” Christine 
said, “but better to have the data, 
right? You wouldn’t launch a new 
product or anew campaign without 

data. HR decisions should be 
approached the same way.” It 
was a pitch that Christine had 
made countless times while 
stumping for the new initiative. 
e “We're confident that decisions 
made using our algorithms 
are reasoned, strong, and less 

biased by personal feelings toward 
employees,” she said. 

Aliyah turned to Brad. “I assume 
you agree?” 

“Of course,” he said, watching for 
Christine’s reaction. “But as a data 
scientist, I also encourage healthy 
skepticism. Our algorithm is brand- 
new. We’ve used it to inform three 
promotion decisions so far, but it's 
too early to tell howthose people 
are doing. I don’t want to give the 
impression that we're 100% confident.” 

Christine looked annoyed. “I 
appreciate your caution, Brad, but 
we’ve heard from the hiring managers 
that the type of recommendations the 
algorithm provides is changing the 
way they think about positions and 
candidates. And we’ve been testing the 
system for months now.” 

Aliyah sighed. “Id trust the 
algorithm more if I understood it 
better.” She knew she wasn’t alone 
in her hesitation: Christine’s team 
had gotten a lot of questions 


about the methodology, despite the 
company wide training sessions. 

“Td be happy to talk more about how 
the algorithm works,” Christine replied, 
“but right now you should focus on the 
two candidates. The point of the system 
isn’t to replace your judgment. The 
aim is to surface qualified people you 
wouldn’t otherwise know about so you 
can make a more informed decision.” 

“Tt’ll help you make a less-biased 
decision too,’ Brad chimed in, “by 
relying more on the data and less on 
gut instinct.” 

Aliyah wondered whether Brad 
thought she was unfairly favoring 
Molly. She worried about that herself 
and cared deeply about making an 
objective decision. Would trusting the 
new system help her do that? 

“But the algorithm’s not completely 
neutral either, right?” she said. 

“You're still relying on information— 
performance reviews, résumés—that 
conceivably has bias baked into it.” 

“Fair point,” Christine conceded, 
“and we've worked hard to control for 
that. But as a data-driven firm, we have 
to extend our approach to the most 
important part of our business: people.” 

“It feels like you’re pushing Ed for 
this position,” Aliyah said. 

“Remember, Ihave to takea 
broader view,” Christine said. “We ran 
analysis to show which high potentials 
are at risk of leaving BBI, and Ed was 
near the top of the list. There is not 
likely to be an opening in beauty 
products, and we want to keep him.” 

“But what about Molly?” Aliyah 
said. “She’ll be devastated if she doesn’t 
get this job, and I’m sure she’d start 
looking too.” 

“Our analysis didn’t flag her as a flight 
risk,” Brad said. “But you could be right.” 


DECISION TIME 


Aweek later, Aliyah wasn’t any closer to 
a decision. She’d been avoiding Molly 
and had put Brad’s analyses in a drawer. 
Ed was clearly qualified, and he’d 
impressed her. But she knew intuitively 
that Molly was ready for the job. 

Did she prefer Molly because of their 
close relationship? Would Molly stay at 
BBI even if she was passed over? 

Aliyah needed to makea 
decision. Should she trust the 
algorithm or her instincts? 


Data scientist Cathy 
O’Neil warns in her 

book Weapons of Math 
Destruction that although 
algorithms are fairly easy 
to create using historical 
data and can improve 
the efficiency of decision 
making, people often 
rely on them without 
understanding the biases 
they may be propagating. 


How does using 
algorithms to analyze 
customers differ from 
using them on employees? 
Should companies 

be more cautious in 
implementing these 
methodologies internally? 


SEE COMMENTARIES ON THE 
NEXT PAGE © 
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CASE STUDY TRUST THE ALGORITHM OR YOU 


SHOULD ALIVAH 
HIRE MOLLY OR ED? 
THE EXPERTS 
RESPOND 


PRASAD SETTY 


R GUT? 


AS AN ENTHUSIAST and practitioner of 
people analytics for many years, I believe 
that it is best applied as a complement 

to, not a substitute for, human judgment. 
For example, an algorithm could be used 
to widen the set of candidates a hiring 
manager might consider for a role. In 

the case of BBI, the people analytics 
system appears to have done a good jobin 
surfacing Ed, an unexpected candidate. 

If the algorithm is going to make the 
promotion decision on Aliyah’s behalf, 
however, the burden of proofis very high. 
BBI’s track record of three decisions isn't 
enough, in my opinion, to demonstrate 
the system’s reliability. When it comes 
to hiring and promotions, it is especially 
important to be able to explain why a 
particular choice is being made. Aliyah is 
right to want more clarity on the system's 
methodology. She needs to know not 
just which candidate the algorithm is 
recommending but on what basis. 

So what would I suggest that Aliyah do? 
She should define exactly what she’s trying 
to achieve in filling the role—something 
the algorithm is not likely to know—and 
base her decision on that. If success 
for Aliyah means bringing a talented 
marketing director on board as soon as 
possible and having that person operate 
at the highest velocity right away, then 
Molly appears to be the better fit. If Aliyah 
is more keen on increasing collaboration 
with other parts of the company, then Ed, 
who has a wider network of relationships, 
seems to be the smarter choice. In either 
case, it's important for Aliyah to own the 
decision and be able to articulate why she 
made a particular choice. 

In an ideal world, BBI would strive 
to make objective, unbiased hiring 
decisions by setting up a structured 

interview protocol and 
evaluation process, 
predetermining 
criteria, and 
defining 
what poor, 


mediocre, and great skills in candidates 
would look like. The company would 
advertise all roles to attract a broad 
applicant pool. Multiple people would 
conduct the evaluation, and the people 
making the final decision would not 
be the same people who evaluated the 
candidates. Analytics could be used to set 
up these protocols, recommend potential 
candidates, and track the impact of 
decisions on on-the-job performance. 
That's not what's happened in this case: 
Aliyah now has to select one of the 
candidates on the basis of BBI’s existing 
process, the information she has, and 
what her goals are. 

At Google, our managers don’t 
make hiring and promotion decisions 
unilaterally. All open positions are 
made public within the organization, 
and anyone is free to apply. We use 
independent committees to assess 
applicants using a structured rubric 


IF THE ALGORITHM IS GOING 
TO MAKE THE PROMOTION 
DECISION ON ALIYAH'S 
BEHALF, THE BURDEN OF 
PROOF IS VERY HIGH. 


detailing what it takes to succeed in the 
job. We analyze the outcomes of these 
processes—for example, whether people 
thrive in their new roles—to ensure that 
we make high-quality people decisions. 
When we started Google’s people 
analytics group, our goal was to make 
all people decisions based on data and 
analytics. In the decade since then, we 
have seen some of the limits of data- 
driven human resources decision. Our 
goal now is to arm leaders with data and 
context so that they feel more confident 
in their choices, but not to undercut their 
role in the process. Today, our team’s 
mantra is to help all people make decisions 
based on data and analytics. 
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PATTY MCCORD 


ALIYAH SHOULD TRUST her instincts, not 
the algorithm. Only she knows what she 
truly needs from a director of marketing, 
and it's clear that Molly hasit. 

Before I left Netflix, I was pitched 
by a lot of data analytics companies 
that promised to help us make better 
people decisions using their algorithms, 
but I didn’t see the ROI. At the time, 
we employed a few thousand people, 
and the cream was already rising to the 
top, so it was hard to imagine how an 
algorithm would significantly improve 
our decisions. BBI is a much bigger 
company, so it’s possible that AI could be 
more useful in that context. But for now, 
the decision should rest with Aliyah. 

Of course, she needs to follow best 
practices for hiring managers and look 
at the team as a whole in her analysis. 
What is its current makeup, and how will 
the newly promoted director fit in? What 
results would Aliyah expect to see in six 
months to a year to demonstrate that the 
division is working well? And with the 
proper support, which candidate is most 
likely to achieve those results? 

Aliyah is also right to question her 
own bias toward a woman she considers 
a protégé and with whom she has worked 
closely for several years. Hiring managers 
often say, “I’m looking for someone who 
is smart, solid on their feet, and ready 
to jump into the role” and it just so 
happens that the person they like best 
looks just like them. This isn’t always 
overt bias. It’s often a matter of not 
wanting to take a risk. 

Although I don’t see any evidence that 
Aliyah is favoring Molly because she’s 
a woman, I do think gender should be a 
consideration in this case. Research has 
repeatedly demonstrated that women are 
not promoted at the same rate as men. 
Throughout my HR career, I’ve seen men 
consider female candidates but hire a 
man instead, telling me, “I know she’s 
qualified, but I’m looking for someone 
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who is ready 
to step up now, 
and I don’t want 
to set her up to fail.” Of course, you can’t 
be set up to win if no one gives you the 
opportunity to fail. I worry that if Molly 
doesn’t get this chance, she might not 
get another. 

Christine touts her algorithm as 
unbiased, but without more information 
about the methodology and the criteria it 
uses, we can’t be sure she’s right. In fact, 
Pm particularly suspicious of the network 
data Brad brought to the meeting. Could 
it be that Ed is more connected—that he’s 
invited to more meetings, in touch with 
more people—because he’s a man and has 
been given more opportunities to shine? If 
I were Aliyah, I’d ask what those network 
maps look like for men and women across 
the company. 


CHRISTINE TOUTS HER 
ALGORITHM AS UNBIASED, 
BUT WITHOUT MORE 
INFORMATION ON THE 
CRITERIA IT USES, WE 

CAN'T BE SURE SHE'S RIGHT. 


Pll admit that if the gender roles were 
reversed in this case, with a male hiring 
manager favoring a male candidate he 
knew well over a woman recommended by 
an algorithm, my advice would probably 
be different. In this case, the algorithm 
has done its job in recommending an 
unexpected man. But if Aliyah has 
conducted a fair analysis—consciously 
setting her preconceptions aside and 
objectively considering who is best for the 
job—and she still prefers Molly, she should 
trust her gut and promote her. @ 
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COMMENTS FROM THE 
HBR.ORG COMMUNITY 


It's Personal 

Algorithms help identify 
potential candidates, but 
they may not be able to 
tell how well an employee 
can integrate into a new 
team. Intangibles such as 
personal chemistry and 
ability to work together 
cannot be determined 
with something as clinical 
as an algorithm. 

Thorfida Charles, 

lead consultant, Reliance 
Professional Systems 

and Services 


Go for Ed 

Not only does the data 
show that Ed is better 
suited for the position 
but he interviewed very 
well. Molly has an unfair 
advantage, and a decision 
to promote her over Ed 
would appear to be 
biased and based on 
personal opinion. 
William Cummings, 
disclose operations executive, 
PR Newswire/Cision 


Give It Time 

| would introduce a rule 
that candidates must 
match a certain percentage 
of the criteria—over, 

say, 80%-—+to qualify for 
consideration. From there, 
the line manager leads the 
decision-making process. 
Algorithms “learn” and 
become more accurate as 
data is fed in. A year down 
the line, this algorithm 
should be more reliable. 
Daniel Vacassin, founder, 
Indigogold 
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